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ABSTRACT 

Three new indicators of psychometric quality for 
objectives-based statewide assessments are proposed. These measures 
provide indication of the stability of reported data on item and 
objectives mastery, the validity of assessment items for members of 
various cultural groups, and the convergent validity of prescribed 
objectives mastery scores. The results provided should also have 
application in situations other than statewide assessments. In 
particular, the results should be applicable whenever the 
psychometric quality of measurements for institutions, rather than 
individuals, is of concern. (Author/EC) 



<3 Sa^E PSYCi!0[';ETRIC INDICATORS FOR STATEWIDE ASSESSMENTS' 

Richard M* Jaeger 
^ University of South Florida 
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LU Prolegomenon 

Strictly speaking, this paper Is true to Its title: It does provide some 
psychometric Indicators for statewide assessments. However Its distribution of 
content Is skewed. In that Indices of stability are given far greater attention 
than are Indices of validity. The results provided should also have application 
In situations other than statewide assessments. In particular, the. results should 
be applicable whenever the psychometric quality of measurements for Institutions, 
rather than Individual ^ Is of concern. 

A review of state accountability legislation reveals that states allege a 
multiplicity of purposes for their assessments (Hawthorne, 1974). Some leglsla- 
tures mandate uniform measurement of all pupils, presumably to provide bases for 

q7 Individual decisions. More often, however, the legislative objectives of assess- 
ment require aggregated Information on pupils In various Institutions— schools, 
school districts, or specific educational programs. The results of these measure- 
ments -e Intended to provide bases for decisions concerning the institutions, 
rather than the Individual pupils thoy serve. 

^JJ^ Examples of legislation that motivate Institutional measurement Include the 

CO Connecticut State Legislature's Public Act Number (1971), that requires the State 
Board of Education to develop an assessment procedure to measure the adequacy and 
effectiveness of educational programs in Connecticut's public schools; Georgia 
^JJ^ State Senate Bill Number 672 (1974), that requires the State Board of Education 

'Presented as part of a symposium on Advances In the Methodology of State- 
S wide Assessment, at the Annual Meeting of the American Educational Research 
Association^ Washington, D.C., April, 1975. 
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to establish performance-based criteria to evaluate the Instructional program of 
each school In the state; and Section 290.1 of the Pennsylvania School District 
Reorganization Act of 1963, -that requires the State Board of Education to develop 
en evaluation procedure for objectively measuring the state's educational pro- 
grams. 

With statev;Ide educational assessment has come Increased attention to tech- 
niques and procedures for measurement. Altnough E. L. Thorndike defined the dif- 
ference between criterion-referenced and norm-referenced measurement In 1918, and 
the Boston Public Schools conducted a criterion-referenced assessment In 1916, both 
the term and the practice are enjoying a renaissance that would make one doubt 
their earlier origins. The current measurement literature abounds with state- 
ments on the relative worth of criterion-referenced and norm- re fere need measure- 
ment, and articles on methods for assessing the reliability and validity of cri- 
terion-referenced moasures are numerous (Stanley, I97I; Livingston, 1972a, 1972b, 
1973; Harris, 1972, 1973; Ebel, 1973; Pop ham and husek, 1969). 

Since many statewide assessment programs attempt criterion-referenced Inter- 
pretations of their measurements, one might think that psychometric Indices for 
such measurements would be sufficient for state assessments. They are not. The 
very concept of reliability, although Inherently generic, has been developed In 
the context of measuring Individuals (Lord and Novtck, 1968, p. 61; Stanley, 1971, 
p. 357; Cronbach, 1951). New reliability formulations Intended for use with cri- 
terion-referenced measures (Livingston, 1972a) have also been proposed as Indices 
of the stability of Individual assessments. Indices of stability for Institution- 
al measures are therefore still to be developed. One could support a similar 
case for Indices of validity. 
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Indicators of Stability 

T he classical definition of reliability ^ Lord and Novick (1968, p. 61) de-^ 
fine the reliability of a test as "a measure of the degree of true-score varia- 
tion relative to observed-score variation/' As Is typical, they refer to varia- 
tion among the true scores and observed scores of Individuals. The Lord and 
Novick definition of reliability can be adapted directly to an index of stability 
for the mean measurement performance of a group of pupils. The formulation is 
as fol lows: 

Consistent with classical test theory, assume that the observed measuremenr 
for the l-th person Is composed of true-score and error terms, 

that errors are uncorrelated with true see; 33, and that errors are uncorretated 
across persons; 



Assume In addition, that the mean Is based upon measures of n^ persons, chosen from 
some larger population through simple random sampling (Cochran, 1963). 

Given these assumptions, the reliability of an Individual's score Is equal 

to 





2,2 

^1 X, T, 




= a J /(a J +a p ) 
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and the "reliability" of the group mean score equals 

2 2 2 2 ^ 

Pxx ^ ^ > - ' - ~ ' Pxx' 

^x 4>E 4 4/n + cTE/n + 



where X = ZX| = £(T^ Ei ) = T + E. 

n n 



As traditionally defined, then, the reliability of the mean score of a ran- 
domly sampled group of persons Is i dentical to the reliability of the score of 
an Individual sampled from the same population. 

Although It may be comforting to be in familiar territory with the reliability 
of group mean scores, the Interpretation of the Index Is not clear. If, for 
example, X represents 1he mean achievement test score of the fourth-graders In a 
single school, one might attempt test-retest estimation of the reliability of the 
mean by randomly sampling schools, administering the test on successive occasions 
to all fourth-graders In each sampled school, and computing the correlation be- 
tween Individuals* successive scores. If thQ sampled schoois could be considered 
representative of schools in some larger administrative unit, or perhaps In the 
nation, and If fourth-graders could be assumed to be randomly allocated among 
schools, the resulting reliability coefficient would be an estimate of the 
stability of school means. The magnitude of the corresponding standard error of 
measurement could be evaluated by considering national norms for school mean 
achievement scores (U.S. Government Printing Office, 1974). To avoid the assump- 
tion of random allocation of pupils to schools, one could compute the ecological 
correlation of school means for successive administrations of the test. How- 
ever, the population value of the resulting coefficient would not then equal the 
reliability coefficient for Individuals. An estimate of reliability consistent 
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wlth the Lord and Novfck definition would result only If true-score means and error 
means were uncorrelated across schools, and rriean errors resulting from successive 
administrations of the test were uncorrelated across schools. Since some school 
environmental factors would likely contribute to error In consistent ways. It Is 
doubtful that the latter assumption could be met. 

A Generalization — Universes and Universe Scores , Rather than defining the 
stability of group mean scores through direct extrapolation of classical test 
theory, It would seem more productive tdrflrst give attention to the meaning de- 
sired In such an Index. The theoretical underprlnnings for the generalization 
proposed here are contained In Cronbach, Gleser; Nanda and Rajaratnum (I972), A 
specific citation provides the needed background: 

"A behavioral measuremrint Is a sample from the collection of measurements 
that might have been made, and Interest attaches to the obtained score only be- 
cause It Is representative of the whole coiiectlon. If the decision maker could, 
he would measure the person exhaustively c:nd take the average over all measure- 
ments." 

"Educators and psychologists have traditionally referred to the average 
reached via exhustlve measurement as *the true score* for the person. We speak 
Instead of a universe score . This emphasizes that the Investigator Is making an 
Inference from a sample of observed data, and also that there Is more than one 
universe to which he might generalize. Any person fits within many defferent 
populations. . . . Any observation fits within a variety of universes." 

If references to persons are replaced with references to Institutions, the 
major concepts In the paragraphs cited above still apply. Any measurement on an 
Institution Is a sample from a population of measurements thet might be made. 
And the Instltutlon(s) measured constitute a sample from several potential popu- 
lations of Institutions. An Index of the stability of measurement of an 
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Institution should specifically reflect the generalizations desired, both with 
re?pect to a population o-^- potential measurements, and to a population of Institu- 
tions. Indices of stability referenced to specific generalizations will facilitate 
unai.iblguous Interpretations. 

A Taxononr<y of Universe Scores for Statewide Ass os sments . Since the objectives 
of statewide assessment vary among the states and a given statewide assessment may 
have several purposes, ft Is not surprising that a number of different universe 
scores could be of Interest. At least five dimensions can be used to structure 
a taxonomy of universe scores and corresponding estimators for statewide assess- 
ments. These dimensions are 

1) The evaluative referent for Interpretation of the universe score: 

a) domain of content or abilities 

b) normative 

2) The type of statistic that constitutes the observed score: 

a) the proportion of examinees that answers a question correctly 

b) the proportion of examinees that answers a subset of questions 
correctly 

c) the proportion of examinees that achieves a given cutoff score 

d) the proportion of questions answered correctly by an examinee 

e) the mean score achieved by an examinee 

f) the percentile rank of a mean on a national norm distribution 

g) the percentile rank of a group percentile, referenced to a national 
norm distribution 

3) The universe of measurement content: 

a) a single question on a measurement Instrument 

b) all questions on the measurement Instrument administered 

c) questions on the measurement Instrument administered that are used 
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to assess mastery of an objective 

d) all questions that could be used to assess mastery of an objective 

e) all questions that could be used to assess status In a content domain 

4) The universe of examinee general Izatfon—al ( examinees of a given age 6r 
grade In ihe administrative unlt(s) designated: 

a) the state 

b) each school system In the state 

c) each school In the state 

d) each classroom In the state 

e) each school in u schoo! system 

f) each classroom In a school system 

5) The procedure used to select examinees for assessment: 

a) measurement of aii examinees in the universe of Interest 

b) measurerrient of a simple ranoorn r>3rnple of examinees* 

c) measurement of a stratified '';jrrple of examinees 

d) measurement of all examinees in a simple random sample of classrooms 

e) measurement of all examinees In a simple random sample of schools 

f) measurement of all examinees In a simple random sample of school 
systems 

Neither the dimensions nor the categories of the taxonomy provided above are 
claimed to be exhaustive. They represent combinations of factors that describe 
assessments conducted In several states during the past four years (e*g*, Pennsyl- 
vania, Florida, Oregon and California) and assessment procedures Judged to be of 
potentia I I nterest. 

A given unlvf^rse score and a corresponding observed score are completely de* 
scribed by selecting a category from each taxonomic dimension. However, some com- 
binations of categories provide universe scores unlikely to be of Interest, and 
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other combinations may be logically Inconsistent. An example of a logically con- 
sistent universe score-observed score combination Is provided by categories l-^>, 
2-a), S-a), 5-b). Here the universe score would be the proportion of exam- 

inees In each school system !n the state that can answer a particular question 
correctly. The Interpretation of this universe score would be referenced to the 
domain of content from which the question was selected. The observed score would 
be a Sample proportion for each school system, based on measurement o< a simple 
random sample of examinees In each school system In the state. Considering only 
combinations of examinee universes and examinee selection procedures that are 
logically consistent and feasible, the taxonomy generates 595 different situations* 

Generalized lndlc3S of Stability . The most widely used Indices of the reli- 
ability of Individual scores follow Spearman^s (1904) definition: "the average 
correlation between one and another of... several Independently obtained series 
of values for p." However, as was lllustrared above, correlations of successive 
observed scores for Institutions may not provide stability Indices that can be 
Interpreted In useful or unambiguous ways. Three alternative Indices of stability 
are suggested here. 

In the literature on sampling from finite populations, the most widely used 
Indicator of the stability of a statistic Is Its standard error; that Is, the 
standard deviation of the sampling distribution of the statistic. If the sampling 
distribution of a statistic Is known (or better yet. If the central limit theorem 
can be Invoked), confidence staiements can be constructed using the value of the 
statistic and Its standard error. If 4> Is a universe score of Interest, and q Is 
an estimator of with a distribution that Is asymptotically normal, an approxi- 
mate lOO(l-ct) percent confidence Interval on f(> Is of the form q ± za 
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where 

; /^z denotes the IOO(l-ct/2) percentile of the standard normal dlstrl- 
I -a/2 

butjon, and o denotes the standard error of q. 

q 

The standard error of the observed score used as an estimator of the universe 
socre Is thus suggested as an Indicator of stability for use In statewide assess- 
ments. 

An advantage of the standard error Is that Its magnitude Is expressed In 
the sarr^e units as those of the observed score It describes. Thus if the observed 
score Is the mean raw score on an achievenont test for a random sample of third- 
graders In a school syrt.m, the standard error of the mean will also be expressed 
In raw-score points. For some purposes, th^s otherwise convenient feature of the 
standard error can bo troublesome. For example, If the stabilities of two measure- 
ment procedures that used different Instruments were to be compared, direct com- 
parison of respective standard errors would not, In general, be appropriate. In 
most Instp.nces, one unit on the scale of measurement of one Instruemnt would not 
equal one unit on the scale of measurement of another Instrument. A useful fea- 
ture of Spearman^s reliability Index Is Its lack of dependence on the units of 
the measurement Instrument It describes. 

An alternative Indicator cf the stability of a statistic that has the "unit-- 
less" property of the Spearman reliability coefficient Is the coefficient of 
variation (cv). The coefficient of variation Is a dlscrlptor sometimes used In 
the theory of sampling from finite populations. It Is equal to the ratio of the 
standard error of a statistic to the value of the statistic. Thus for an observed 
score q with standard error o^, the coefficient of variation equals 

cv(q) - ^q/q* 

The coefficients of variation of the observed scores on two different measure- 
ment Instruments can be directly compared, without reference to the units of 

10 
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measurement of either instrument. The larger the coefficient of variation, the 
less stable the estimate of -the universe score of Interest. 

To be consistent with 1he traditions of reliability estimation. It may be 
desirable to have an Index of stability, rather than an Indicator of the Instab- 
ility of an observed score. Such ai; Index can be resally constructed from the 
coefficient of variation as follows: 

Daflne the Index of stabi llty (IS) of an observed score q, used as an 
estimator of a universe score 4^ , to be 

IS(q) = [ I - cv(q)3lOO= C 1 - a^/q3lOO percent. 

Using this definition, the Index of stability of an observed score equals 
100 percent only If the standard erroi* of the score, across all elements of the 
universe, equals zero. The Index of stability equals zero If the standard error 
of the observed score Is equal In magnitude to the observed score (Note that each 
of the observed-score statistics listed In dimension 2 of the taxonomy given above 
can only assume nonnegattve values.) The Index of stability assumes negative 
values only when the standard error of an observed score Is larger than the value 
of the observed score. 

. Universe Scores, Observed Scores and their Estimated Standard Errors . Each 
combination of factors In the taxonomy provided above leads to a universe score, 
a corresponding observed score, and a standard error of the observed score. In 
order to estimate the stability of an observed score using the Indices suggested 
In the preceedlng section, an estimate of the standard error of each type of 
observed score must be available. 

The combinations of universes of examinee generalization, examinee sampling 
procedures, observed scores, and universes of measurement generalization that are 
logically consistent and likely to be of some Interest In a statewide assessment 
provide 595 entries In the previously described taxonomy. If the suggested Indices 



ERIC 



-II- 

of stability are to be computed, an estimator (formula for computing an observed 
score) and an estimate of standard error Is needed for each of these entries. Al- 
though estimators and standard errors are nearly Identical for some entries In 
the taxonomy, examination of all 595 cases Is beyond the scope of this paper. 
Only the 170 cases generated by observed score entry 2a) "The proportion of exam- 
inees that answers a question correctly" and entry 2b) "The proportion of examin- 
ees that answers a subset of questions correctly" have been Investigated. 

An Index to estlma**-ors of universe scores and corresponding standard errors 
Is provided In Table I. Entries In this table define components of estimators, and 
reference specific formulas provided In Table 2» As an example, suppose the 
evaluative referent for Interpretation of the universe score of Interest Is a 
comain of content or abilities (Category la), the type of statistic that consti- 
tuted the obsen^'ed score Is the proportion of examinees that answers a subset of 
questions correctly (Category 2b), -the universe of measurement content Is all 
questions ^n the measurement Instrument administered (Category 3b), the universe 
of examinee generalization Is composed of all examinees of a given age or grade 
In the state (Category 4a), and the procedure used to select examinees Is simple 
random sampling (Category 5b). Reference to the appropriate estimator of the 
universe score of Interest, and the standard error of the estimator, can then be 
found In Table 1 as Equation (lO)* Additional Information In Table I, needed to 
use Equation (10) for the specified purpose. Includes the following definitions: 
N denotes the size of the examinee population In the state, n denotes the size 
of the examinee sample, M denotes the number of questions on the measurement 
instrument to which general l7atlon Is desired, and m denotes the number of ques- 
tions sampled from the measurement Instrutnent* 

After determining the appropriate universe-score estimator and s^andard 
error for a given purpose, the user would turn to Table 2 to find the needed 
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foHiiuIas. Tho pcifametor:- of formulas given In Table 2 would be defined using 
the Information given !n the appropriate cell of Table I. In the example under 
discussion, th3 estimator of the unlvarse score of Interest Is the sample 
proportion of examinee-question contacts that result In a correct answer (p»f/nm), 
where f Is the number of exsm I nee-quest I on contact? •*''^+ result In a correct 
answer. The standard error of p Is given by the , assfon that followo S(p). 

No derivations of estimators or standard errors are provided In this paper. 
Soino of the re<:ults provided are new (particularly for stratified sampling of 
examln?ies), but most have been adapted and extended from the writings of 
Jaeger (1970), Lo^d and N'ovlck (!968, Chsr/ler II), and SIrotnil< (i974). 
Detailed derivation of .nany ro5uMT> Involving mrtrix sampling may be fo>'nd In 
Lord and Novick, and SirotnlK derive:^ esUmators of standard errors for several 
matrix sampling problems. Derivations of many results Involving one-dimenslona I 
sampling (sarnpMng either examinees or questions, but not both) follow develop- 
ments provided In Cochran (1963). 

In all cases where generalizations from a subset of questions to a sub- 
suming set of questions Is to be made. It Is assumed that elements of the 
subset are selected through simple random sampling. When generalizations 
are to be made to "all questions that could be written," It Is assumed that 
the universe of questions Is Infinite In size. In situations Invoh'ing 
stratified sampling of examinees. It Is assurriCd that examinees are selected 
using Independent simple random sampling procedures within each stratum. 
Finally, In situations Involving cluster sampling (e.g., sampling of classrooms, 
schools or school systems). It Is assumed that clusters are selected using 
simple random sampling, that clusters may be unequal In size, and that all 
examinees within a sampled cluster are measured (single-stage cluster sampling). 
The estimators suggested for use In cluster sampling situations provide unbiased 
estimation of the universe scores of Interest, and other than cluster sizes, 
require no Information on the populations within cK^ters. Alternative single- 
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stage cluster sampling and estimation procedures may provide moro efficient 
estimation of universe scores, but these procedures require auxiliary Information 
on examinees In the population (see Jaeger, 1970, 1973 for details). 

For a number of the situations Identified In Table I, no analytic solution 
for the standard error of the estimated universe score Is known. For these 
cases, the reader Is referred to Table 2 for an unbiased estimator of the 
universe score of Interest, and Is Instructed to use the jackknife procedure 
to estimate the standard error. Details on the application of the Jackknife 
procedure can be found In Miller (1964), Mortal |er and Turkey (1968, and 
Jaeger (1970). 

Indicators of Cultural Validity 

An achievement measure can be terfr.3d "culturally valid" provided groups 
or Individuals of equal ability have the same chance of performing successfully. 
Cole (1973) proposed a philosophically similar definition of tests free from 
bias, when used for purposes of selection* Alternative definitions of culturally 
fair tests have been provided by Thorndtke (1971) Darlington (1971), Elnhorn 
and Bass (1971), and Linn (1973). 

Since the measures used In statewide assessments are administered to 
Individuals from diverse cultural and racial groups, and to groups of diverse 
cultural and racial composition, cultural validity (In the sense of being 
free from cultural or racial bias) Is Important. 

To apply the definition of cultural validity proposed above, one must 
have a culturally-fair measure of ability to use as a standard. In statewide 
assessments (as In many other measurement situations), such culturally-fair 
ability measures are unlikely to exist. Thus It may be Impossible to determine 
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the cultural validity of an entire measurement Instrument In these applications* 
However, It Is feasible to determine the degree to which components of a 
measurement Instrument provide relative cultural validity, provided performance 
on the entire Instrument Is accepted as a standard of ability. Cardall and 
Coffman (1964) and Cleary and Hilton (1968) exa^nlned the relative cultural 
blab of items on particular measurement Instruments (the Scholastic Aptitude 
Test and the Preliminary Scholastic Aptitude Test, respectively) using Type IN 
analysis of variance models. They used the Item-by-group Interaction component 
of variance ns an Indicator of the degree of relative cultural bias In the 
collective items of an Instrument. The Cardall and Cleary procedures did not 
permit the Identification of specific Irems that contributed to relative 
cultural bias; they only provided a global Indicator of relative bias for the 
entire Instrument. 

Another procedure for estimating the relative cultural validity In the 
components of a measurement Instrument wss proposed by Angoff and Sharon 
(1974). Their method provided Indicators of relative cultural validity for 
each Item, In addition to providing an overall Index for the Instrument. 
Angoff and Sharon computed a normalized transformation of Item difficulties 
for members of each of two cultural groups, used these to construct a scatter 
plot, and computed the equation of the major axis of the elliptical envelope 
surrounding the scatter of points. For each Item, the perpendicular distance 
from the major axis was used as an Indicator of relative cultural bias. 
Both the analysis of variance procedure and the Angoff and Sharon procedure 
might be said to employ the definition of cultural validity proposed here, 
but each uses the score on an entire measurement Instrument as a measure of 
ability. 
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A I though the Angoff and Sharon procedure was proposed for use with a 
norm-referenced achievement test, It could be applied to Item? that compose a 
domain-referenced or objectives-referenced measure. In fact, the procedure 
should 'provide Interpretable results when used with any set of Items that are 
homogeneous In' content or purpose. 

An alternative Indicator of the relative cultural validity of Items that 
compose a measure can be developed using empirical Item characteristic functions 
(Henrysson, 1971). Again, the Indicator Is phi losophica I ly consistent with 
Cole's conditional probability model (1973) of culture fairness. Examinees' 
scores on the entire achievement measure are used as a surrogate for a culture- 
fair ability measure. The resulting sacrifice is an absolute Index of cultural 
validity; only an Index of relative validity can be obtained. 

If the relative cultural validity of an Item for members of two groups 
is to be estimated (call the groups Group A and Group B), proceed as follows. 
For each group, compute the cumulative distribution of the proportion of exam- 
inees who are successful on the Item of interest, as a function of total score 
on Items in a subsuming content domain (or on a subsuming test). For example, 
suppose a domain-referenced measure of ability to recognize sound blends 
contains 10 Items, and the relative cultural validity of an Item requiring 
identification o* the "bl" sound blend is to bfi determined. The cumulative 
proportion of examinees from Group A who successfully identify "bl" and who 
earn scores of zero, 6ne or less, two or less, . . ., on the entire sound 
blend Item sample would be computed, (this is an empirical item characteristic 
function). The same procedure would be completed for examinees In B. As a 
measure of the relative cultural validity of the "bl" Item, the maximum 
difference between the two empirical Item characteristic functions would be 
computed. The smaller the maximum difference, the greater the relative cultura 
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va Hdlty of the Item. Fictitious Item characteristic functions are Illustrated 
In Figure I , below. 




0 r 2 3 '4 5 6* 7 8 '9 10 

Score on Items In consonant blend sample 
F igure I: Fictitious Empirical Item Characteristic Functions 



In the example portrayed by the graphs In Figure I, the maximum difference 
between the Item characteristic functions occurs at a score value of 5, and 
equals about 30 percent* In this example then, the "bl" Item would have very 
low relative cultural validity, being relatively biased against members of 
Group A. Inspection of the curves would show that members of Group A who 
correctly answer about 50 percent of the items sampled from the consonant 
blends domain suffer the greatest relative bias on the "bl" Item. 

The procedure for esthnating relative cultural validity proposed here 
would appear to identify those Items that are relatively Invalid, as does the 
Angoff and Sharon (1974) procedure, and in addition. Identify the overall 
performance level of group members who suffer the greatest relative bias. 
This additional information may be useful when one seeks to determine why some 
Items show relative cultural bias. 

Procedures for Investigating Criterion Validity 

In some statewide assessments, a pupil Is said to have mastered an objective 
If {s>he successfully answers k out of K Items related to that objective. 
The Items related to a specific objective are said to constitute an exercise. 
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The problem of establishing a valid mastery score (k) Is herein called the 
"criterion validity" problem. 

In a recent review of research, Mlllman (1973) describes five procedures 
for establishing mastery scores on objectives-referenced exercises* Several of 
the procedures appear to be so administratively cumbersome as to be Impractical, 
and all require subjective Judgment at some point In their application. In short, 
there appears to be no universally best solution to this problem. 

The procedure proposed here also depends upon subjective Judgments, but 
Identifies exercises for which Independent Judgments are Inconsistent. 
Its strength then, lies In Its consistency requirements. 

For random samples of several hundred pupils In each grade assessed, 
teachers could be asked to specify whether or not each pupil has achieved 
mlnlrrial mastery In each conten+ domain assessed. Data should be collected In 
such a way that teachers' Judgments of minimal nastery can be matched to pupils' 
performance on assessment exercises. 

In addition to providing these data, a small sample of randomly selected 
teachers should be asked to subjectively Judge the difficulty of exercises 
associated with each objective In +he domains assessed In their grades. 
These teachers would be asked to p»"ovIde two types of Judgments. First, they 
would be asked to estimate the proportions of pupils who have and have not 
achieved mastery of the content domain who should successfully answer I out of 
K, 2 out of K, . . ., K out of K Items used to assess mastery of an objective. 
Second, these teachers would be asked to relate the difficulty of the exercises, 
by estimating the proportion of pupils who should be able to master a given 
objective, given that they have demonstrated mastery of another objective. 
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Using these data, tha consistency of actual performance and estimated 
performance on each exercise could be examined- The proportions of pupils 
Identified as masters and non-masters (a) that can successfully answer I out 
. of 2 out of K, • • K out of K Items In an exercise and (b) that teachers 
judge should be able to successfully answer I out of K, • • , , K out of K 
items in an exercise, could be compared. Large discrepancies between actual and 
judged percentages will pinpoint exercises for which prescribed mastery levels 
are Inconsistent with Independent judgments. 

Teachers judgements of the conditional proportions of pupils who should 
exhibit mastery of one objective, given mastery of another, can be compared to 
actual proportions. Again, inconsistencies will pinpoint exercises for which 
mastery levels should be reconsidered. 

Finally, considering once again the prospect that exercises can be 

classified into logical content domains, one could examine criterion validity by 

analyzing responses to all exercises In a content domain. If there are N ex- 

E 

erclses in a content domain, the proportion of pupils who have achieved mastery 
of a given exercise in the domain, and who have achieved mastery of I out of 
exercises, 2 out of exercises, . . out of exercises In the domain, 

can be tabulated. Any deviation from a rronoton I ca 1 1 y Increasing function would 
indicate inconsistency in prescribed mastery levels. 

Summary 

The most frequent interpretations of data from statewide assessments are 
Institutional, rather than Individual. To judge the quality of such interpreta- 
tions. Indices of stability and validity appropriate for Ins. itutlonal assess- 
ments are needed. Just as traditional psychometric indices prescribe upper 
bounds on the quality of interpretations of Individual measurements, appropriate 
for institutional measurements will provide much needed warning flags. 
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Th Is paper provides several Indices of psychometric quality for 
institutional Interpretations of measurement likely to be found in statewide 
assessments. Hov/ever, the situations for wnlch indices of stability are 
provided, and the limited types of measurement validity considered mai<e but 
a small dent In a problem of major scope. Standard errors of observed scores 
must be developed for the 425 cells of the taxonomy not considered in this 
paper. And certainly, the taxonomy presented does not exhaust the situations 
that may arise in statewide assessments or other Institutional uses of 
measurement. In the area of validity, progress has barely begun. As the 
purposes of statewide assessment are more clearly delineated, needs for 
validity indices parallel to those used with individual problems — the yalldlty 
of domain representations, predictive validity, concurrent validity — will ccme 
Into focus. 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES* 



1) tvaluatlve Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A single question 
on a measurement 
i nstrument 


b) Al 1 questions on 
the measurement 
Instrument admin- 
istered 


4) Universe of Examin- 
ee General izatton 


5) Selection 
Procedure 


a) The State 


a) All exar nees 
in state 


Standard Error 
equals zero 


Standard error Is not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (6). 
N=pop. size In state 
M=No. questions on 
meas. Instrument 


b) SRS examinees 
In state 


Equation ( i ) 

N=pop. size In state 

n=sample size In 
state 

p=sample proportion 
that answer cor- 
rectly 


Standard error Is not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (8). i 
N=pcp. size In state 
n=examlnee sample size 
M=No. questions on 
meas. Instrument 


c) Stratified 
sample of 
examinees In 
state 


Equation (2) 
N=pop. size In state 
K=No. of strata 
Nj^= pop. size In 

stratum k 
n=sample size 
P|^=sample propor- 
tion In stratum 
k 


Standard error Is not 
estimable unless al 1 
questions are of equal 
difficulty for examin- 
ees within a stratum. 
This assumption Is 
probably untenable In 
most cases. 



*See Table 2 for numbered estimators of standard errors. 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


Z) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A Single question 
on a measurement 
i nstrument 


b) At 1 questions on 
the measurement 

Jtrument admin- 
istered 


4) Universe of Examin- 
ee General izat I on 


5) Selection 
Procedure 


a) The State 


d) SRS class- 
rooms In 
state 


Equation (3) 

N=No. classrooms In 

state 
n=No. classrooms in 

samp le 
M =pop. examinees 

in state 
classroom-cluster 


Standard error is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equatjon (3) to 
estimate p and jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms in 

state 
n=No. classrooms In 

sample 
M^spop. examinees 

In state 
classrx)om=cl uster 


e) SRS schools 
In state 


Equation (3) 
N=No. schools in 

state 
n-No. schools in 

samp le 
M^=pop. size ex- 
ami nees 1 n state 
school =c luster 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard • 
error. 

N=No. schools in 

state 
n=No. schools In 

sample 
M^=pop. size ex- 
aminees In state 
school=c luster 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question 5s randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A single question 
on a measurement 
Instrument 


b) Al 1 questions on 
the measurement 
Instrument admln- 
I ste red 


4) Universe of Examin- 
ee General I Eat I on 


5) Selection 
Procedure 


a / 1 n© oTaT© 


T/ DIaO 5cnooi 

systems In 
state 


Equation (3) 
N=No, systems In 

state 
n=No, systems In 

samp le 

0 * 

ami nees I n state 
school system=cl uster 


Standard error is not 
estimab le un less 
questions are of 
equal difficulty; 
then analytic sol- 

1 1+ 1 n -for Q+a nHa rH 

error Is unknown, so 
use Equatjk)n (3) to 
estimate p and jack- 
knife procedure to 
estimate standard 
error, 

N=No, systems in 
st3te 

n=:Mn ^v^+fim^ I n 
1 1 IHKJ • s y s 1 Wilis 1 1 1 

samp le 
M =pop. size ex- 
^ ami nees in state 
schoo 1 sys tem=c 1 u ste r 


b) Each school system 
In state 


a) Al 1 examin- 
ees in school 
system 


Standard Error 
equals zero 


Standard error is not 
estimable unless 
questions are of equal 
difficulty; then use 
Enuat 1 on (6 ) . 
N=pop. size In system 
M=:No. questions on 
n»*ias. Instrument 


b) SRS examinees 
In school 
system 


Equation (1) 

N=pop. size In 
system 

n=samp le size in 
system 

p=sample proportion 
that answer cor- 
rectly 

1 


Standard error Is not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (8)» 
N=pop. size in system 
n=examlnee sample size 
M=No. questions on 
meas. Instrument 
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Table I 

DESIGNATIONS OF ESTIMATORS OF STANDAku ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A single question 
on a measurement 
instrument 


b) Ai 1 questions on 
the measurement 
Instrument admin- 
istered 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


b) Each school system 
In state 


c) Stratified 
samp le of ex- 
ami neos in 
school system 


Equation (2) 
N=pop. size in 

system 
K=No. strata 
N.=pop. size in 

stratum k 
n=samp le size 
p.=sampie propor- 
tion in stratum 
k 


Standard error is not 
estimab le unless al 1 
questions are of equal 
difficulty for examin- 
ees within a stratum. 
This assumption Is 
probably untenable in 
most cases. 


d) SRS class- 
rooms in 
school system 


Equation (3) 

N=No. classrooms in 

system 
n=No. classrooms In 

samp le 
M^=pop. size examin- 
ees in system 
c 1 OSS room=c 1 uste r 


Standard error is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

system 
n=No. classrooms in 

samp le 
Mq= pop. size examin- 
ees in system 
c 1 assroom=c 1 uster 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDAP*^ urtRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A single question 
on a measurement 
I nstrument 


b) Al 1 questions on 
the measurement 
I nstrument adml n- 
Istered 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


h) Pach <?chool svstem 

W 1 WU^I 1 Owl \\J\J 1 O V O 1 VIII 

In state 


e) SRS schools 
i n system 


Equation (3) 
N=No. schools in 

system 
n=N'o. schools In 

samp 1 e 
M =DOD • s I ze ex^ 

ami nees In system 
school =c luster 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equatjjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate stano rd 
error, 

N=No. schools in 

system 
n=No- school s I n 

II 1 • W 1 1 ^^^J 1 W III 

sample 
M =pop. size examln- 
^ ee? In system 
school =c luster 


c) Each school In 
state 

or 

e) Each school in 
a system 


a ) A 1 1 exam I nees 
In school 


Standard Error 
equals zero 


Standard error Is not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (6), 
N=pop. size In school 
M=No. questions on 
meas. I nstrument 


b) SRS examinees 
I n school 


Equation ( 1 ) 

N=pop. size In 
school 

n=sample size In 
school 

p=sarriple propor- 
tion In school 
that answer cor- 
rect 1 y 


Standard error Is not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (8). 
N=pop. size In school 
n=examlnee sample size 
M=No. questions on 
meas. Instrument 
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Table I 



OESIGNATrONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A single question 
on a measurement 
Instrument 


b) Al 1 questions on 
the measurement 
Instrument admin- 
istered 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


c) Each school In 
state 

or 

e) Each school In 
a system 


c) Stratified 
sample of 
examinees in 
school 


Equation (2) 
N=pop, size In 

school 
K=No. of strata 
N. =pop, size In 
^ stratum k 
n=sample size 
P(^= sample propor- 
tion In stratum 
k 


Standard error Is not 
estlmab le unless al 1 
questions are of equal 
difficulty for examin- 
ees within a stratum. 
This assumption Is 
probably untenable In 
most cases. 


d) SRS class- 
rooms In 
school 


Equation (3) 

N=No. classrooms In 

school 
n=No. classroom^ In 

samp le 
M =pop, exam I nees 
^ In school 
c 1 ass room=c 1 uster 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution Is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

school 
n=No. classrooms In 

samp le 
M =pop. exami nees 
^ In school 
c I ass room=c luster 


d) Each classroom 
In state 
or 

f) Each classroom 
In a system 


3) Al 1 examinees 
In classroom 


Standard Error 
equals zero 


Standard error fs not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (6). 
N~pon size In class- 
r A 

M=Nv . questions on 
meas. Instrument 
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Table I 



desigjniations of estimators of standard errors of observed scores 
used as estimators of universe scores 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


a) A single question 
on a measurement 
I nstrument 


b) Al 1 questions on 
the measurement 
Instrument admin- 
istered 


4) Universe of Exam In-- 
ee General Izatlon 


5) Selection 
Procedure 


d) Each classroom 
In state 
or 

f) Each classroom 
In a system 


b) SRS examinees 
in classroom 


Equation ( 1 ) 

N=pop. size in 
classroom 

n=samp le slzft! 1 n 
classroom « ■ 

p=sample proportion 
that answer cor- 
rect 1 y 


Standard error Is not 
estimable unless 
questions are of equal 
difficulty; then use 
Equation (8)« 
N=pop. size In class- 
room 

n=examlnee sample size 
M=No. questions on 
meas. instrument 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


c) Questions used 
to assess mastery 
of an object I ve 


d) Questions thct 
could be used to 
assess mastery of 
an objecti ve 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


a) The State 


a) AM examinees 
In state 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then use Eauatlon 
(6). 

N=:pop. size In state 
M=No. questions that 
oertal n to ob- 

J 1 1 vo 


Standard error Is not 
estimable unless al I 
questions in popu- 
lation are of equal 
difficulty; then use 
Equation (7). 
N= pop. size In state 


5) SRS examinees 
In state 


Standard error Is not 
estimable unless 
questions are of 
eaual difficulty; 
tne.; use Equation 
(8). 

N=pop. size In state 
n=exam!nee sample 
s ! ze 

M=No. questions that 
pertai n to ob- 
jective 


Standard error Is not 
estimable unless al 1 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (9). 
n=examinee sample size 


c) Stratified 
sample of 
examinees In 
state 


Standard error Is not 
estimable unless al 1 
questions are of 
equal difficulty for 
exam I nees wl thi n a 
stratum. This as- 
sumption Is prob- 
ably untenable In 
frost cases. 

■ 


Standard error Is not 
estimable unless al I 
questions are of equal 
difficulty for examin- 
ees withl n a stratum. 
This assumption Is 
probably untenable in 
most cases. 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



\) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
.question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


c) Questions used 
to assess mastery 
of an objective 


d) Questions that 
could be used to 
assess mastery of 
an object Ive 


4) Universe of Examin- 
^ee General Izatlon 


Selection 
Procedure 


a) The State 


d) SRS cl ac;Q- 
rooms I n 
state 


Standard error is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ut ion for standard 
error Is unknown, so 
use EquatjjDn (3) to 
estimate p and jack- 
knife procedure to 
estimate standard 
error. 

N=No, classrooms In 

state 
n=No. classrooms In 

samp le 
Mq=POp. examinees 

I n state ■ 
c 1 ass room=c 1 uste r 


Standard error Is not 
estimable un less 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use EquatjjDn (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

state 
n=No. classrooms in 

samp le 
M -pop. examinees 

In state 
c 1 ass room=c 1 uste r 


e) SRS schools 
In state 


Standard error is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error is unknown, so 
use EquatjjDn (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. schools In 

state 
n=No. schools In 

samp le 
M^=pop. size ex- 
aminees in state 
school=cl uster 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. schools fn 

state 
n=No. schools In 

sample 
M^=pop. size ex- 
aminees I n state 
school=c luster 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


c) Questions used 
to assess mastery 
of an objective 


d) Questions that 
couid be used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


a / \ n© OTOT© 


systems In 
state 


Standard error is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 

ii+l on "for ^+anHarH 

error Is unknown, so 
use EquatJjDn (3) to 
estimate p and Jack- 
knl fe procedure to 
estimate standard 
error. 

N=No, systems In 

state 
n=No, systems In 

sample 
M =pop* size ex- 
^ ami nees In stste 
schoo I system=c 1 uster 


Standard error Is not 
estlmab le unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equation (3) to 
estimate p and Jack- 
kni fe procedure to 
estimate standard 
error. 

N=No. systems In 

state 
n=No. systems In 

sample 
M =pop. size ex- 
^ ami nees In state 
schoo 1 systemic I uste r 


b) Each school system 
in state 


a) Al 1 examin- 
ees In school 
system 


Standard error Is not 
estimable unless 
questi ons are of 
equal difficulty; 
then use Equation 
(6). 

N=pop. size In 

system 
M=No. questions 

that pertain to 

objective 


Standard error Is not 
estlmab le unless al 1 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (7). 
N=pop. size In system 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCARES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities .f 


2) Type of Statistic - 
Observed Scx^re 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


c) Questions used 
to assess mastery 
of an object I ve 


d) Questions that 
could be used to 
assess mastery of 
an obJectI ve 


4) Universe of Examin- 
ee Genera 1 izatlon 


5) Selection 
Procedure 


b) Each school system 
In state 


b) SRS exami nees 
In school 
system 


Standard error is not 
estimable unless 
questions are of 
equal dl f f Iculty; 
then use Equation 
(8). 

N=pop. size In 

system 
n=examinee samp le 

s I ze 
M=No. questions 

that pertain to 

objecti vo 


Standard error Is not 
estlmeble unless al 1 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (9). 
n=examlnee sample 
s Ize 


c) Stratified 
samp le of ex- 
am I nees 1 n 
school system 


Standard error fs not 
estimable unless a! 1 
questions are of 
equal difficulty for 
examinees wl thi n a 
stratum. This as- 
sumption Is probably 
untenable In most 
cases. 


Standard error Is not 
estimable unless al 1 
questions are of 
equal difficulty for 
examinees within a 
stratum. This as- 
sump.tlon Is probably 
untenable In most 
cases. 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



I) Evaluative Referent 



^) lype of Statistic - 
Observed Score 



a) Domain of content or abilities 



a) Proportion of examinees that answer a 
single question correctly (Assume 
question is randomly sampled from rele- 
vant universe) 



3) Universe of 

Measurement Content 



4) Universe of Examin- 
ee General Izatlon 



5) Selection 
Procedure 



c) Questions used 
to assess mastery 
of an objective 



d) Questions that 
could be used to 
assess mastery of 
an objective 



b) Each school system 
In state 



d) SRS class- 
rooms I n 
school system 



Standard error is not 
estimable unless 
questions are of 
equal difficulty; 
then ana lytic sol- 
ution for standard 
error Is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure t 
estimate standard 
error. 

N=No. classrooms In 

system 
n=No. classrooms In 

samp I e 
M^=pop. size examin- 
ees In system 
c I ass room=c luster 



Standard error Is not 
estimable unless 
questions are of 
equa! difficulty; 
then analytic sol- 
utions for standard 
error Is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

system 
n=No, classrooms In 

samp I e 
M =pop, size examin- 
ees In system 
class room=c I uste r 



e) SRS schools 
In system 
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standard error Is not 
ostimab le unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error, 

N=No. schools In 

system 
n=No. schools in 

samp I e 
M =pop. size examin- 
ees In system 
schoo I =c luster 



not 



Standard error Is 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. schools In 

system 
n=No. schools In 

sample 
M =pop. size exam I n- 
^ ees i n system 
schoo I =c luster 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



I) Evaluative Referent 



2) Type of Statistic - 
Observed Score 



a) Domain of content or abilities 



a) Proportion of examinees that answer a 
single question correct !y (Assume 
question Is randomly sampled from rele- 
vant universe) 



c) Questions used 
to assess mastery 
of an objective 



d) Questions that 
could be used to 
assess mastery of 
an objective 



3) Universe of 

Measurement Content 



4) Universe of Examin- 
ee General izatlon 



c) Each school In 
state 

or 

e) Each school In 
a system 



5) Selection 
Procedure 



a) All exami nees 
In school 



Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then use Equation 
(6). 

N=pop. size in 

school 
M=No. question^ hat 

pertal n to ou- 

jectl ve 



not 



Standard error Is 
estimable unless 
questions are of 
equal difficulty; 
then use Equation 
(7). 

N=pop. size In schoo 



b) SRS examinees 
In school 



Stanoard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then use Equation 
(8). 

N=pop. size In 

school 
n=examlnee samp le 

size 

M=No. questions that 
porta In to ob- 
jectl ve 



Standard error 1 5 not 
estimable unless al I 
questions I n popu- 
lation are of enual 
difficulty; then use 
Equation (9). 
n=examlnee sample size 



c) Stratified 
samp le of 
examinees in 
school 



Standard error Is not 
estlmab ie unless al I 
questions are of 
equal difficulty for 
examinees within a 
stratum. This as- 
sumption Is probably 
untenable I n most 
cases. 



Standard error Is not 
estimable unless a 1 1 
questions are of 
equal dl ff Icuity for 
examinees within a 
stratum. This as- 
sumplton is probab ly 
untenable In most 
cases. 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of examinees that answer a 
single question correctly (Assume 
question Is randomly sampled from rele- 
vant universe) 


3) Universe of 

Measurement Content 


c) Questions used 

assess mastery 
of an objective 


d) Questions that 
could be used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee Genera 1 Izatlon 


5) Selection 
Procedure 


c) Each school I n 

o 1 a 1 o 

or 

e) Each school in 
a system 


d) SRS class- 

ronrriQ t n 

school 


Standard errorTs not 
estlmab le unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution Is unknown, so 
use Equation (3) to 
estimate ^ and jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

school 
n=No. classrooms in 

sample 
M^=pop* examinees 

In school 
c I ass room=c I uste r 


Standard error 's not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution Is unknown, so 
use Equatjon (3) to 
estimate p and jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

school 
n=:No. classrooms In 

sample 
M =pop. examinees 
^ In school 
class room=c I uste r 


d) Each classroom 
In state 
or 

f) Each classroom 
In a system 
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a) All examl nees 
In classroom 


Standard error Is not 
estlmab le unless 
questions are of 
equal difficulty; 
then use Equation 
(6). 

N=pop. size In class- 
room 

N'=N'o. questions that 
pertain to ob- 
jective 


Standard error Is not 
estimable unless al 1 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (?)• 
N=pop. site In class- 
room 


b) SRS examinees 
In classroom 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; • 
then use Equation 
(8). 

N=pop. slze^ In class- 
room 

n=examtnee sample 
size 

M=No. questions that 
pertain to ob- 
jective 


Standard error Is not 
estimable unless al I 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (9). 
n=examlnee sample size 



Table I 



DESIGNATIONS OF ESTimTORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTimTORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of ex- 
ami nees that an- 
swer a single 
question correctly 
(Assume question 
Is randomly sam- 
pled from relevant 
un I verse) 


b) Proportion of ex- 
ami nees that an- 
swer a subs,et of 
questions correct- 
ly (Assume simple 
random samp 1 1 ng 
of questions) 


3) Universe of 

Measurement Content 


e) Questions that 
could be used to 
assess status In 
a content domain 


a) A single question 
on a measurenr,ent 
instrument 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


a) The State 


a) All exami nees 
in state 


Standard error Is not 
estimable unless al 1 
questions In popu- 
lation are of equal 
dl f f icul ty; then use 
Equation (7) . 
N=pop» size in state 


Standard error Is not 
estimable unless 
sped f Ic question Is 
sampled; then standard 
error equals zero. 


57 SRS examinees 
In state 


Standard error is not 
estimable unless all 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (9). 
n=examinee sample 
size 


Standard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation ( 1 ). 
N=pop. size In state 
n=sample size In state 
p=sample proportion 
that answer cor- 
rectly 


c) Stratified 
sample of 
examinees in 
state 


Standard error is not 
estimable unless al 1 
questions are of 
equal difficulty for 
examinees within a 
stratum. Thls.as- 
sumptlon Is probably 
untenable 1 n most 
cases 


Standard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation (2). 
N=pop. size In state 
K=No. of strata 
N|^=pop. size In 

stratum k I 
p =sample proprotlon ! 

^ In stratum k j 

1 

1 
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rable I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of ex- 
aminees that an- 
swer a single 
question correctly 
(Assume question 
Is randomly sam- 
pled from relevant 
universe) 


b) Proportion of ex- 
aminees that an- 
swer a subset of 
questions correct- 
ly (Assume simple 
random samp! Ing 
of questions) 


3) Universe of 

Measurement Content 


e) Questions that 
could be used to 
assess status In 
a content domain 


a) A single question 
on a measurement 
i nstrument 


4) Universe of Examin- 
ee General Ization 


5) 


Selection 
Procedure 






a) The State 


d) 


SRS class- 
rooms 1 n 
state 


Standard error is not 
estimable unless 
questions are of 
eaual difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

state 
n=No. classrooms In 

samp le 
M =pop. examinees 
° In state 
c 1 ass room=c 1 uste r 


Standard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation (3). 
N=No. classrooms In 

state 
n=No, classrooms In 

sample 
M =pop. examinees In 
° state 

c I ass room=c 1 uste r 
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e) 


SRS schools 
in state 


Standard error Is not 
estimable un less 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use EquatjjDn (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error, 

N=No. schools In 

state 
n=No. schools In 

sample 
Mq=Pop. size ex- 
aminees In state 
school =cl uster 


Standard error Is not 
estimable unless 
sped f Ic question Is 
sampled; then use 
Equation (3). 
N=No. schools In state 
n=No. schools In 

samp le 
M =pop. examinees In 
° state 

school=c luster 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



I) Evaluative Referent 



a) Domain of content or abilities 



Type of Statistic - 
Observed Score 



a) Proportion of ex- 
aminees that an- 
swer a single 
question correctly 
(Assume question 
Is randomly sam- 
pled from relevant 
universe) 



3) Proportion of ex- 
am Inees that an- 
swer a subset of 
questions correct- 
ly (Assume simple 
random samp 1 1 ng 
of questions) 



3) Universe of 

Measurement (Content 



e) Questions that 
could be used to 
assess status In 
a content domain 



a) A single question 
on a measurement 
I nstrument 



4) Universe of Examin- 
ee General Izatlon 



a) The State 



5) Selection 
Procedure 



Standard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation (3)- 
N=No. systems In 

state 
n=No, systems In 

sample 
'\ =pop. size ex- 
° aminees In state 
school system=c luster 



) SRS school 
systems I n 
state 



Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unknown, so 
use Equation (3) to 
estimate p and jack- 
knife procedure to 
estimate standard 
error. 

N=No. systems In 

state 
n=No. systems in 

samp I e 
M =pop. size ex- 
^ aminees In state 
schoo ! system=c I uste r 



b) Each school system 
in state 



a) AM examin- 
ees in school 
system 



Standard error Is not 
estimable unless al I 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (7). 
N=pop. size In system 



Standard error Is not 
estimable unless 
specific question is 
sampled; then standard 
error equals zero. 



b) SRS examinees 
In school 
system 



Standard error is not 
estimable unless al I 
questions In popu- 
lation are of equal 
dl ff Iculty; then use 
Equation (9). 
n=examlnee sample 
size 
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StaTidard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation ( I ) . 
N=pop. size In system 
n=sample size in 

system 
p=sample proportion 
that answer cor- 
rect I y 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of ex- 
aminees that an- 
swer a si ngie 
question correctly 
(Assume question 
Is randc.nl y sam- 
pled from relevant 
universe) 


b) Proportion of ex- 
aminees that an- 
swer a subset of 
questions correct- 
ly (Assume simple 
random sampi Ing 
of questions) 


3) Universe of 

Measurement (intent 


e) Questions that 
could be used to 
assess status In 
a content domal n 


a) A single question 
on a measurement 
I nstrument 


4) Universe of Examin- 
ee Genera 1 Izatlon 


5) Selection 
Procedure 


b) Each school system 
In state 


c) Stratified 
samp le of ex- 
am Inees In 
school system 


Standard error is not 
estimable un less al 1 
questions are of 
equal dl f f iculty for 
exami nees wi thi n a 
stratum. This as- 
sumption Is probably 
untenable In most 
cases . 


Standard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation (2). 
N=pop. size In school 

system 
K=No. of strata 
N.=pop. size In 

stratum k 
p ^sample proportion 
In stratum k 


d) SRS class- 
rooms I n 
school system 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution for standard 
error Is unl<nown, so 
use Equatjon (3) to 
estimate p and Jack- 
knife procedure to 
estimate standard 
error. 

N=No. classrooms In 

system 
n=No. classrooms In 

samp le 
M =pop. si 20 examln- 
^ ees 1 1 system 
c 1 ass room=c 1 uste r 


Standard error Is not 
esttmab le unless 
specific question Is 
samp led; then use 
Equation (3). 
N=No. classrooms In 

system 
n=No. classrooms In 

samp le 
M spop. size exami n- 

ees in system 
classroom=cl uster 



ERIC 



« 



Table I 



DESIGNATION? OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
US'iD AS ESTIMATORS OF UNIVERSE SCORES 



1 ; tvaiuattve Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Obsarved Score 


a) Proportion of ex- 
ami nees that an- 
swer a single 
question correctly 
(Assume question 
is randc!^!y sam- 
pled from relevant 
un I verse ) 


b) Proportion of ex- 
aminees that an- 
swer a subset of 
questions correct- 
ly (Assc -e simple 
random sampi Ing 
of questions) 


3) Universe of 

Measurement Content 


e) Questions that 
could be used to 
assess status In 
a content domain 


a) A single question 
on a measurement 
instrument 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


b) Each school system 
In state 


e) SRS schools 
in system 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
than analytic sol- 
ution for standard 
error Is unknown, so 
use Equatjon (3) to 
estin.ate p and jack- 
knife procedure to 
estimate standard 
error. 

N=No. schools in 

system 
n=K'o. schools In 

samp le 
^"o^pop* size examin- 
ees In system 
school ^cluster 


Standard errcr Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation (3), 
N=:No, schools In 

system 
n=No. schools In 

samp le 
M =pop. size examI n- 

ees 1 n system 
sctiool=cl uster 


c) Each school In 
state 

or 

e) Each school In 
a system 


a) Al 1 exami nees 
In school 


Standard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then use Equation 
(7). 

N=pop. size In 
school 


Standard error Is not 
estimable unless 
specific question Is 
sampled; then standard 
error equals zero. 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abl litles 


2) Type of Statistic - 
Obse rved Sco re 


a) Proportion of ex- 
aminees that an- 
swer a single 
question correctly 
(Assume question 
is randomly sam- 
pled from re levant 
universe) 


b) Proportion of ex- 
aminees that an- 
swer a subset of 
questions correct- 
ly (Assume simp le 
random sampi ing 
of questions) 


3) Universe of 

Measurement Content 


e) Questions that 
could be used to 
assess status i n 
a content domal n 


a) A single question 
on a measurement 
i nstrument 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


' — ^ — r- I r 1 — r~i ■ 

c) Each school In 
state 

or 

e ) Each schoo 1 I n 
a system 


b) SRS examinees 
In school 


Standard error is not 
estimable unless a 1 1 
questions In popu- 
lation are of equal 
difficulty; then use 
Equation (9). 
n=exami nee samp le 
size 


Standard error Is not 
estlmab le unless 
sped f Ic question* Is 
sampled; then use 
Equation ( 1 ). 
N=pop« size In school 
n=sample size in 

school 
p=sample proportion 
that answer cor- 
rectly 


c) Stratified 
sample of 
examinees in 
school 


Standard error is not 
estimable unless a 1 1 
questions are of 
equal difficulty for 
examinees within a 
stratum* This as- 
sumption Is probably 
untenab le in most 
cases. 


Standard error Is not 
estimable unless 
specific question is 
sampled; then use 
Equation (2). 
N=pop, size In school 
K=No. of strata 
N =pop. size In 

stratum k 
p ^sample proportion 

In stratum k 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USEO AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


a) Proportion of ex- 
aminees that an- 
swer a single 
question correctly 
(Assume question 
Is randomly sam- 
pled from relevant 
universe) 


b) Proportion of ex- 
aminees that an- 
swer a subset of 
questions correct- 
ly (Assume simple 
random sampi fng 
of questions) 


3) Universe of 

Measurement Content 


e) Questions that 
could be used to 
assess status In 
a content domain 


a) A single question 
on a measurement 
Instrument 


4/ Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


c) Each school In 
state 

or 

e) Each school In 
a system 


d) SRS class- 
rooms In 
school 


^.tandard error Is not 
estimable unless 
questions are of 
equal difficulty; 
then analytic sol- 
ution Is unknown, so 
use EquatJ[on (3) to 
estimate p and jack- 
knife procedure to 
estimate standard 
error, 

N=No. classrooms In 

school 
n=No. classrooms In 

samp le 
M =pop. examinees 

In school 
classroom=cl uster 


Standard error Is not 
estimable unless 
specific question Is 
sampled; the^'i use 
Equation (3). 
N==No. classrooms In 

schoo 1 
n=No. classrooms In 

samp le 
M =pop. examinees In 
^ school 

c 1 ass room=c 1 uste r 


d) Each classroom 
In state 
or 

f) Each classroom 
In a system 

ERLC 


a) All examinees 
In classroom 


Standard error Is not 
estimable unless al 1 
questions I n popu- 
lation are of equal 
difficulty; then 
use Equation (7). 
N=pop. size In class- 
room 


Standard error Is not 
estimable unless 
specific question Is 
sampled; then standard 
error equals zero. 


b) SRS examinees 
In classroom 


Standard error Is not 
estimable ur less al 1 
questions In popu- 
lation are of equal 
dl ff Iculty; then use 
Equation (9). 
n=examlnee sample 
size 
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Standard error Is not 
estimable unless 
specific question Is 
sampled; then use 
Equation ( 1 ) 
N=:pop. size In class- 
room 

n=:sample size In 
c 1 ass room 

pssample proportion 
that answer cor- 
rectly 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic 


b) Propbrrlon of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


b) Al 1 questions on 
the measurement 
Instrument admin- 
istered 


c) Questions used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


a) The State 


a) Al 1 examinees 
In state 


EquatlcJPl (4) 

M=No. questions 
on nieas. Instru- 
ment 

m^No. questions 
sampled 

N=pop. si 7/5 In 
state 


Equation (4) 

M=No. questions on 
Instrument that 
pertain to objective 

m=No. sampled 

questions that per- 
tain TO OD J ecT i ve 

N=pop, size In state 


b) SRS examinees 
In state 


Equation (10) 
N=pop. size In state 
n=examlnee sample 
size 

M=No. questions on 

I nstrument 
m=No. questions 

samp 1 ed 


Equation (10) 
N=pop. size in state 
n=examlnee sample 
size 

M«No. questions on 
Instrument that per- 
tain to objective 

m=No. sampled 

questions thct per- 
tain to objective 


c) Stratified 
sample of 
exam i nees i n 
state 


Ana lyti c sol utlon Is 
unknown unless 
questions are sam- 
pled Independently 
for each examinee 
stratum; then use 
Equation (12). 
N • =nnn ^ 1 76 I n 

stratum I 
K=No. of strata 
n.=sample size In 

' straium I 
l'=No. questions on 

1 nstrument 
m=No. questions 

sampled for each 

st ratum 


Analytic solution is 
unknown unless 
questions are sam- 
pled Independently 
for each examinee 
stratum; then use 
Equation (12). 
N =DOD . s I ze I n 

stratum I 
K=No. of strata 
n =sample size In 

' stratum I 
M=No. questions on 
Instrument that 
pertain to ob- 
jective 
m=No. sampled 

questions that per- 
tain to objective 
for each stratum 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1/ tvaiuaTive KeterenT 


3) uomain ot conTenT or ao in Ties 


Z) Type of Statistic - 
Observed Score 


b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


b) At 1 questions on 
the measurement 
I nstrument adml n- 
Istered 


c) Questions used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee General Ization 


5) Selection 
Procedure 


a) The State 

ERIC 


d) SRS class- 
rooms in 
state 


Analytic solution for 
standard error Is un- 
known. Use Equation 
(3) to compute p, 
then use Jackknlfe 
procedure to esti- 
mate standard error, 
N=No, classrooms In 

state 
n'=No, classrooms In 

samp 1 e 
M^=pop, examinees 

In state 
classroom=cl uster 


Analytic solution for 
standard error Is un- 
known. Use Equation 
(3) to compute p, 
then use jackknlfe 
procedure to esti- 
mate standard error. 
N=No. classrooms In 

state 
n=No, classrooms In 

samp le 
M =pop. examinees 
° In state 
cIassroom=cIuster 


e) SRS schools 
In state 


Analytic solution fo/ 
standard error Is un- 
known. Use Equation 
(3) to compute p, 
then use Jackknlfe 
procedure to esti- 
mate standard error, 
N=No, schools In 

state 
n=No, schools in 

samp le 
M =pop, examinees 
0 in state 
schooled uster 


ifnalytic solution for 
standard error Is un- 
known. Use Equation 
(3) to compute p, 
then use Jackknlfe 
procudure to esti- 
mate standard error. 
N=No. schools In 

state 
n=No, schools In 

sample 
M =pop. exami nees 

In state 
schooled uster 


f) SRS school 
systems i n 
state 

45 


Analytic solution for 
standard error Is un- 
known • Use Equation 
\j) TO compuTo p, 
then use Jackknlfe 
procudure to esti- 
mate standard error, 
N=Mo. systems In 

state 
n=No. systems In 

samp le 
M =pop, examinees 
^ in state 

school systemic! uster 


Analytic solution for 
standard error Is un- 
known. Use Equation 
\j J TO compuTe p ^ 
then use Jackknlfe 
procudure to estl^ . 
mate standard error, 
N=No. systems In 

state 
n=No. systems In 

sample 
M =spop, examinees 

In state 
school systemic luster 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



I) Evaluative Referent 



2) Type of Statistic - 
Observed Score 



3) Universe of 

Measurement Content 



4) Universe of Examin- 
ee General 1 zation 



i5) Selection 
Procedure 



b) Each school system 
In state 



a) All exam! n- 
^ ees In school 
system 



d) SRS examinees 
\i\ school 
system 



ERLC 



a) Domain of content or abilities 



b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 



b) Al I questions on 
the measurement 
instrument admin- 
istered 



c) Stratified 
sample of ex- 
aminees In 
school system 



Equation (4) 

N=pop. size In school 
system 

M=No. questions on 
meas. i nstrument 

i=Mo. questions sam- 
pled 



c) Questions used to 
assess mastery of 
an objective 



Equation (10) 

N=pop. size In school 

system 
n^examlnee sample 

size 

M»No. questions on 
Instrument 

m=No. questions sam- 
pled 



Analytic solurlon Is 
unknown unless 
questions are sampled 
Independently for 
each examinee 
stratum; then use 
Equation (12). 
N.=pop. size In 

' stratum I 
K=No. of strata 
n.=sample size In 

stratum I 
M=Mo. questions on 

instrument 
rri=No. questions sam< 

pled for each 

stratum 



Equation (4) 

N=pop. size In school 

system 
M=No. questions on 
Instrument that per- 
tain to object We 
m=No. sampled 

questions that per- 
tain to objective 



Equation CIO) 

N=pop. size In school 

system 
n-examlnee sample 

size 

M=No. questions on 
Instrument that per- 
tain to objective 

m=No. sampled 

questions that per- 
tain to objective 



Analytic solution Is 

unknown unless 

questions are sampled 

Independently for 

each examinee 

stratum; then use 

Equation (12). 

N -pop. size In 
* stratum I 

K=No. of strata 

n =samp le size In 
stratum I 

M=No. questions on 
Instrument that per- 
tain to objective 

m=:No. sampled 

questions that per- 
tain to objective 
for each stratum 



Tat le I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
vjDservea ocor© 


b) Proporrlon of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


b) Al 1 questions on 
the measurement 
Instrument admin- 
istered 


c) Questions used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


b) Each school system 
In state 


d) SRS class- 
rooms I n 
school system 


Analytic solution for 
standard error Is un- 
known. Use Equat^lon 
(3) to estimate p, 
then use Jackknife 
procedure to estimate 
standard error. 
k'=Mo. classrooms In 

system 
n=No* classrooms In 

samp le 
M =pop* examinees In 
° system 

c 1 as5room=c 1 uster 


Analytic solution for 
.standard error Is un- 
known. Use Equat^Ion 
(3) to estimate p, 
then use Jackknl fe 
procedure to estimate 
standard error* 
N=No. classrooms In 

system 
n=No, classrooms In 

samp 1 e 
M =pop. examinees In 
^ system 

cl assroom=cl uster 


e) SRS schools 
In system 


Analytic solution for 
standard error Is un- 
known. Use Equat^Ion 
(3) to estimate p, 
then use Jackknife 
procedure to estimate 
standard error. 
N=No. schools In 

system 
n=No. schools In 

samp 1 e 

M =pop. examinees 
or 1 
In sample 

school=c luster 


Analytic solution for 
standard error Is un- 
known. Use Equat^lon 
(3) to estimate p, 
then use Jackknife 
procedure to estimate 
standard error* 
N=No. schools In 

system 
n=No. schools In 

samp le 
M =pop. examI nees 
^ 1 n samp 1 e 
school =cl uster 


c) Each school In 
state 

or 

e) Each school In 
a system 


a) All exami nees 
In school 


Equation (4) 
N=pop. size In 

school 
M=No. questions on 

moas. Instrument 
rn=No, questions 

samp led 


Equation (4) 

M=No. questions on 
Instrument that per- 
tain to objective 

m=No. sampled 

questions that pe*^- 
taln to objective 

N=pop, size In school 



ERIC 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abl lltles 


2) Type of Statistic - 
Observed Score 


b) Proportion of examlmees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


b) Al 1 questions on 
the measurement 
Instrument admin- 
istered 


c) Questions used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


c) Each school In 
state 

or 

e) Each school in 
a system 


b) SRS examinees 
in school 


Equation (10) 
N=pop. size in school 
n=examlnee sample 
size 

M=No. questions on 

Instrument 
m-No. questions 

samp 1 ed 


Equation (10) 
N=pop. size In school 
n=examlnee sample 
size 

M=No. questions on 
Instrument that per- 
tain to objective 

m=No. sampled 

questions that per- 
tain to objective 


c) Stratified 
samp le of 
examinees In 
school 


Analytic solution Is 
unknown unless 
questions are sam- 
pled Independently 
for each examinee 
stratum; then use 
Equation (12). 
N =pop. size in 

stratum I 
K=No. of strata 
n =sample size in 

' stratum I 
M=No. questions on 

Instrument 
m=No. questions 

sampled for each 

stratum 


Analytic solution Is 
unknown unless 
questions are sam- 
pled Independently 
for each examinee 
stratum; then use 
Equation (12). 
N =pop. size In 

stratum t 
K=No. of strata 
n =samp le size In 

stratum 1 
M=No. questions on 
Instrument that per- 
tain to objective 
m=No. sampled 
questions that per- 
tain to objective 
for each stratum 



ERIC 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


b) Al 1 questions on 
the measurement 
1 nstrument admin- 
istered 


c) Questions used to 
assess mastery of 
an objective 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


c) Each school In 
state 

or 

e) Each school In 
a system 


d) SRS class- 
rooms In 
school 


Analytic solution for 
standard error Is un- 
known. Use Equat^Ion 
(3) to estimate p, 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. classrooms In 

school 
n=No. classrooms In 

samp 1 e 
M =pop. examinees In 
^ school 

classroom=c luster 


Analytic solution for 
standard error Is un- 
known. Use Equat^Jon 
(3) to estimate p, 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. classrooms In 

school 
n=No. classrooms In 

sample 
M =pop. examinees In 
^ school 

c 1 ass roomie 1 uste r 


d) Each classroom 
In state 
or 

f) Each classroom 
In a system 


a) At 1 examinees 
In classroom 


Equation (4) 
N=pop. size In 

classroom 
M=No. questions on 

mea'3. Instrument 
rn=No. questions 

sampled 


Equation (4) 

M=No. questions on 
instrument th2t per- 
tain to objective 

m=No. sampled 

questions that per- 
tain to objective 

N=pop. size In class- 
room 


b) SRS examinees 
In classroom 


Equation ( 10) 
M=pop. size In 

classroom 
n=examlnee sample 

size 

M=No. questions on 
I nstrument 

m^^No. questions 
samp led 


Equation (10) 
N=5pop. size In class- 
room 

n=examlnee sample size 
M=No. questions on 
Instrument that per- 
tain to objective 
m=No sampled 
questions that per- 
tain to objective 



ERIC 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent t 


a) Domain of content or abilities 


2) Type of Statistic • 
Observed Score 


b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


d) Questions that 
could be used to 
assess mastery of 
an objective 


e) Questions that 
could be used to 
assess status In 
a content domain 


4) Universe of Examin- 
ee General Izat Ion 


5) Selection 
Procedure 


a) The State 


a) Al 1 examinees 
In state 


Equation (5) 

Question population 

assumed Infinite 

m=No. sampled 
questions that per- 
tain to objective 

N=pop. size in state 


Equation (5) 
Question population 
assumed Infinite 
m=No, sampled 

questions 
N=pop. Size In state 


b) SRS examinees 
In state 


Equation <ll) 
N"Pop. size In state 
n^cxamlnee sample 

s ize 
m=No. sampled 
questions that per- 
tain to objective 
Question population 
assumed Infinite 


Equation (11) 
N=pop. size In state 
n=examlnee sample 

size 
m=hto. sampled 

questions 
Question population 
assumed Infinite 


c) Stratified 
sample of 
exami nees 1 n 
state 


Analytic solution Is 

unknown unless 

questions are sampled 

Independently for 

each examinee 

stratum; then use 

Equation (I3)« 

Nppop. size In 
stratum 1 

K=No. of strata 

n =sample size In 
stratum 1 

m=No. questions 
sampled for each 
stratum that per- 
tain to objective 


Analytic solution Is 
unknown unless 
questions are sampled 
Independently for 
each examinee 
stratum; then use 
Equation (13). 
N =pop. size tn 

' stratum 1 
K=No. of strata 
n|=sample size In 

stratum 1 
m=No. questions 

sampled for each 

stratum 



ERIC 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



\) Evaluative Referent 


a) Domain of content or abl 1 Itles 

U r 1 1 lU f 1 1 W 1 W W III W III W 1 U^lllll W w 


2) Type of Statistic - 
Observed Score 


b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


Universe of 
Measurement Content 


d) Questions that 
could be used to 
assess master/ of 
an objective 

• 


e) Questions that 
cou Id be used to 
assess status In 
a content domain 


4) Universe of Examin-' 
ee General Izatlon 


5) Selection 
Procedure 


a) The State 

c 


d) SRS class- 
rooms In 
state 


Analytic solution for 
standard error Is un- 
known. Use Equation 
(3) to compute p, 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. classrooms In 

state 
n=No. classrooms In 

samp le 
M =pop. examinees 
° In state 
c 1 ass room=c 1 uste r 


Analytic solution for 
standard error Is un- 
known. Use Equation 
vj; to connpute p, 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No# classrooms in 

state 
n=No. classrooms in 

sample 
M =pop. examinees 

in state 
c 1 ass room=c luster 


e) SRS schools 
In state 


Analytic solution for 
standard error Is un- 
known. Use Equation 
\0) TO compuTe p , 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. schools In 

state 
n=No. schools In 

sample 
M^=pop. examinees 

in state 
school=cluster 


Analytic solution for 
standard error Is un- 
known. Use Equation 
KD) to compute p, 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. schools In 

state 
n=No. schools In 

sample 
M =spop. examinees 
° in state 
school=c luster 


f) SRS school 
systems In 
state 

51 


Analytic solution tor 
standard error Is un- 
known. Use Equation 

then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. systems In 

state 
n=No. systems In 

sample 
M =pop. examinees 

In state 
school system=c luster 

1 


Analytic solution for 
standard error is un- 
known. Use Equation 

1 X 1 ^^mf\t i4*a f\ 
\JJ TU UUllipU 1 0 \J f 

then use Jackknife 
procedure to esti- 
mate standard error. 
N=No. systems In 

state 
n=l^. systems in . 

samp le 
M^=pop. examinees 

In state 
school systemic luster 
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Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
UDserveu ocore 


b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


d) Questions that 
could be used to 
assess mastery of 
an objective 


e) Questions that 
could be used to 
assess status In 
a content domain 


4) Universe of Examin- 
ee General Izatton 


5) Selection 
Procedure 


b) Each school system 
In state 


a) At I examin- 
ees In school 
system 


Equation (5) 

Question population 

assumed Infinite 

m=No, sampled 

questions that per- 
tain to objective 

N=pop. size In 
school system 


Equation (5) 
Question population 
assumed Infinite 
m=No. sampled 

questions 
N=pop. size In school 

system 


b) SRS examinees 
In school 
system 


Equation (II) 
Question population 
assumed Infinite 
N=pop. size In school 

system 
n=examlnee sample 

size 
m=No. sampled 

questions that per- 
tain to objective 


Equation (II) 
Question population 
assumed Infinite 
N=pop, size In school 

system 
n=examlnee sample 

size 
m=No. sampled 

questions 


c) Stratified 
sample of ex- 
aminees In 
school system 


Analytic solution is 

unknown unless 

questions are sampled 

1 nuepenoenT ly tor 

each examinee 

stratum; then use 

Equation (13). 

Nj=pop. size In 
stratum 1 

K=No. of strata 

n =sample size In 
stratum 1 

m-No. questions 
sampled for each 
stratum that per- 
tain to objective 


Analytic solution Is 
unknown unless 
questions are sampled 
1 nooponaQn T 1 y tsji 
each examinee 
stratum; then use 
Equation (I3), 
Nj«pop, size In 

stratum 1 
K=No. of strata 
n|=sampte size In 

stratum 1 
m=No. questions 

sampled for each 

stratum 
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Table i 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


b) Proportion of examinees that answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


d) Questions that 
could be used to 
assess mastery of 
an objective 


e) Questions that 
could be used to 
assess status In 
a content domain 


4) Universe of Examin- 
ee General Izat Ion 


5) Selection 
Procedure 


b) Each school system 
In state 


d) SRS class- 
rooms In 
school system 


Analytic solution for 
standard error Is un- 
known. Use Ec. at^lon 
(3) to estimate p, 
then use Jackknife 
procedure to estimate 
standard error. 
N=No. classrooms In 

system 
n=No. classrooms In 

samp I e 
M^spop. examinees In 

system 
c 1 assroom=c 1 uster 


Analytic solution for 
standard error Is un- ■ 
known. Use Equat^lon 
(3) to esUmate p, 
then use Jackknife 
procedure to estimate 
standard error. 
N=^k). classrooms In 

system 
n=No. classrooms In 

sample 
M =pop. examinees In 
^ system 

cl assroom=cl uster 


e) SRS schools 
In system 


Analytic solution for 
standard error Is un- 
known. Use Equat^lon 
(3) to estimate p, 
then i»se Jackknl fe 
procedure to estimate 
standard error. 
N=No. schools In 

system 
n=No. schools In 

samp Id 
M^=pop. examinees 

In sample 
school=cl uster 


Analytic solution for 
standard error Is un- 
known. Use Equat^lon 
(3) to estimate p, 
then use Jackknife 
procedure to estimate 
standard error. 
N=No. schools In 

system 
n=No. schools In 

samp 1 e 
M =pop. examinees 

In sample 
schooled uster 


c) Each school In 
state 

or 

e) Each school In 
a system 


a) AM exami nees 
In school 


Equation (5) 

Question population 

assumed Infinite 

m=No. sampled 
questions that per- 
tain to objective 

N=pop. size In 
school 


Equation (5) 
Question population 
assumed Infinite 
m=No. sampled 

questions 
N=pop. size in 

school 

— 



ERIC 



53 



Table I 



DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF OBSERVED SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



1) Evaluative Referent 


a) Domain of content or abilities 


2) type of Statistic - 
Observed Score 


b) Proportion of examinees that answer a 
subset of questions correctly (Assuto 
simple random sampling of questions) 


3) Universe of 

Measurement Content 


d) Questions that 
could be used to 
assess mastery of 
an objective 


e) Questions that 
could be used to 
assess status In 
a content domain 


4) Universe of Examin- 
ee General izat Ion 


5) Selection 
Procedure 


c) Each school In 
state 

or 

e) Each school In 
a system 


b) SRS examinees 
In school 


Equation (II) 
Question population 
assumed Infinite 
N=pop. size In school 
n=exam i nee samp 1 e 

size 
m=N'o. sampled 

questions that per- 
tain to objective 


Equation (11 ) 
Question population 
assumed Infinite 
N=pop. size in school 
n=examlnee sample 

size 
m=No* sampled 

questions 


c) Stratified 
sample of 
examinees fn 
school 


Analytic solution Is 
unknown unless 
questions are sam- 
pled Independently 
for each examinee 
stratum; then use 
Equation (13). 
N.=:pop. size In 

stratum 1 
K=No. of strata 
nj=sample size In 

stratum 1 
m=No. sampled 
questions for 
each stratum that 
pertain to ob- 
jective 


Mnalytic solution Is 
unknown unless 
questions are sam- 
pled Independently 
for each examinee 
stratum; then use 
Equation (13). 
N =pop. size In 

stratum 1 
K=No* of strata 
n =sample size In 

stratum 1 
m=No* questions 

sampled for each 

stratum 


d) SRS class- 
rooms In 
school 


Analytic solution for 
standard error Is un- 
known. Use Equajhlon 
(3) to estimate p, 
then use jackknife 
procedure to esti- 
mate standard error. 
N=No. classrooms In 

school 
n=No. classrooms In 

samp le 
M =pop. examinees in 
^ school 

classrocm-cluster 


Analytic solution for 
standard error Is un- 
known. Use Equat^lon 
(3) to estimate p, 
then use Jackknife 
procedure to esti- 
mate standard error. 
N=No* classrooms In 

school 
n=No. classrooms In 

sample 
M =pop. examinees In 
^ school 

classroom=c luster 
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DESIGNATIONS OF ESTIMATORS OF STANDARD ERRORS OF 0BSERVT.0 SCORES 
USED AS ESTIMATORS OF UNIVERSE SCORES 



\i Evaluative Referent 


a) Domain of content or abilities 


2) Type of Statistic - 
Observed Score 


b) Proportion of examinees tliat answer a 
subset of questions correctly (Assume 
simple random sampling of questions) 


3) Universe of 

Measurement Con'^^'^.t 


d) Questions that 
could be used to 
assess mastery of 
an objective 


e) Questions that 
could be used to 
assess status In 
a content domain 


4) Universe of Examin- 
ee General Izatlon 


5) Selection 
Procedure 


d) Each classroom 
In state 
or 

f) Each classroom 
In system 


a) All examinees 
In classroom 


Equation (5) 

Question population 

assumed Infinite 

m=Mo, sampled 
questions that per- 
tain to objective 

N=pop. size In 
classroom 


Equation <5) 
Question population 
assumed Infinite 
m=No. sampled 

questions 
N=pop. size In 

classroom 


b) SRS examinees 
In classroom 


Equation (ID 

Question population 

assumed Infinite 

N=pop. sIZQ In 
classroom 

n=examlnee sample 
size 

m=No> sampled 
quest! oni; that per- 
tain TO objective 


Equation (II) 
Question population 
assumed Infinite 
N-pop. size In 

classroom 
n=examlnee sample 

size 
m=No. sampled 

questions 
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Table 2 

ESTIMATORS OF UNIVERSE SCORES AMD THEIR STANDARD ERRORS 

ESTIMATOR STANDARD ERROR 

(I) f = number sampled who answer ' S(p) ^ ' (N-n) p( l-p); 

correctly |(n-l)N ' 

( p = f/n ) = (estimator) 



(2) = number sampled in stratum k S(p) = 
who answer correctly 
n|^ = sample size in stratum k 



n2 (n, - I.) 



^ P = J|<^/N>k ^ = (estimator) 



(3) M. = number of examinees in i-th S^p) 
cl uster 

Pj = proportion of successes in 
i-th cluster 

T = 1 Z M.p. 
n i=| II 



! 

n ( l-f ) Z (M.p - T)2j 
M n(n-l) i=l ' ' ! 
o J 



f -•= n/N 



(p = N _g M.p.) = (estimator) 



(4) f = number of sampled question- S(p) =j NM-Nm pd-p) 

examinee contacts that result [(Nm-I )NM 

in comect answers 
Nm = number of sampled question- 
examinee contacts 
UA = population size of quest ion- 
exam i nee coi.'*acts 



1/2 



( p = f/Nm ) = (estimator) 



5f> 



1 



Table 2 

ESTIMATORS OF UNIVERSE SCORES AND THEIR STANDARD ERRORS 



ESTI MATOR STANDARD ERROR 

(5) f = number of sampled question- S(p) = jPjliP) 

examinee contacts that result ^Nm - I 

in correct answer**? 
Nm = number of sampled examinee- 
question contacts 
(p = f/Nm) = (estimator) 



(6) f = number of sampled question- S(p) = 

examinee contacts that result 
in correct answer 
(p = f/N) = (estimator) 



NM - N p(l-p)'} 
(N-l)r^N J 



i 



1/2 



(7) f = number of sampled question- S(p) =i pH-p) 

examinee contacts that result N-l 
in correct answer • 
(p = f/N) = (estimator) 

(8) f = number of sampled question- S(p) =| "Jl P^'^P^ 

examinee contacts that result iTn-l)Nlv1 » 

in correct ansv/er 
(p = f/n) = (estnmator) 



1/2 



(9) f = number of sampled question- S(p) 
examinee contacts that result 
in correct answer 
(p = f/n) = (estimator) 



p(l-p)l ^/^ 
[n-l ] 



ERJC 57 



I 



Toble 2 

ESTIMATORS OF UNIVERSE SCORES AND THEIR STANDARD ERRORS 



ESTIMATOR 

(10) f = numbor of sampled question- 
examinee contacts that result 
in correct answer 
(p = f/nm) = (estimator) 



S(p) 



STANDARD ERROR 

I l(M-m) nvN-na^ + 

^mn(M-l )(N-I ) ' ^ 

m(M-l )(N-n)d2 + (M-m)(N-n)a^j'j|l/^ 



v/here variance component estimators are 
defined as fol lows: 



Let: 



E. = total score of i-th examinee 

on m sampled items 
Ij = total of scores of n sampled 
examinees on j-th item 
score of i-th examinee on j-th 
item 



X. . = 



Define; 
V_ 



i:s| 

m 

n 



I Se? - K 5 E.) 

m(n-l ) [i = l n i = l ' 

I j - i( 

n(m-l ) !j = |J m 



1 



(n-i )(m-| ) 



n n 2 n' 2 

Z Z X. . - I 2 E. 

i=l j=l ^ m 1=1 ' 



m 2 n 2 

1 IT + J. ( Z E. ) 
n j = l J nm i = l ' 



1 



Then; 



3 = N^i. 
^ N 



M 




dpi = (N-l )(M-I ) Vp. 

m 
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Table 2 

ESTIMATORS OF UNIVERSE SCORES AND THEIR STANDARD ERRORS 



ESTIMATOR 

(II) f = number of sampled question- 
examinee contacts that result 
in correct answer 
(p = f/nm) = (estimator) 



S(p) 



mn(M-l ) 



STANDARD ERROR 

[n(N-l + m(N-n)a| + 



h/2 



where variance component estimators 
are defined as follows: 



Let: 



Ej, Ij, Vg, Vj, and V^y be defined 
as in Equation (10). 



Table 2 

ESTIMATORS OF UNIVERSE SCORES AND THEIR STANDARD ERRORS 



ESTIMATOR 



STANDARD ERROR 



(12) f. = number of sampled question- S(p) = 



examinee contacts that result 
in correct answer for examinees 
in i-th stratum. 

f . 



i=l ' ' ! 



N = 



Hjm 
K 



W. = N, 
K 

(p = E V/.p;) = (estimator) 
1 = 1 ' ' 



where 



S(Pj) 



mn. (M-l )(N.-I ) 



j^(M-m)n. (N.-l )d-j^ 



+ m(iv1-l )(Ni-n| )dp. 

I I t, 



+ (M-m)(N.-n. )6,,^, 
I I (EI) 



and the variance components for the i-th 
stratum are defined as follows: 



Let: 



E = total score of j-th examinee 
J in l-th stratum on m sampled 
i terns . 

I = tola I of scores of n. sampled 
examinees in i-th stratum on 
k-th item 

X = score of j-th examinee in i-th 
stratum on k-th item 



Define 
V 



1^2 
I c 2 _ 



. n7E.^i("i E..)l 

riTlj = l 'J n, J = l 'J 



V. = J. S, - 
M "l M n.(m-l) 



I 



(EI) 



i (n.-l)(m-l) 
I 



'? 1 M( § I.,) 
k=l ik m k=l 'k 

i m 2 |"i , 2 
E. Z X, ., -i i E, . 
j= k=l 5jk mj= ij 



1 Sl.S J- ("i E,.) 
nj k=l "k n^m J=l ij 
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Table 2 

ESTIMATORS OF UNIVERSE SCORES AND THEIR STANDARD ERRORS 



ESTIMATOR 



Then: 



a = Ni - 



STANDARD ERROR 



- (l-m/M)V 



<EI): 



M 



1: 



m 

(i-"hv 

NT 



(EI), 1 



n. 



0,^,, = (N, - I )(M - I ) 



(13) f . = number of sampled quest ion- 

* examinee contacts that result 
In correct answer fot <a<o : 
examinees in the 1-th stratum. 

p, =/ 



N = 



i,N. 
1 = 1 i 



W. = N. 
I I 



(p = ^ W p,) = (estimator) 
1 = 1 i 1 



S(p) 



= ri\wy(p,) 



i = l i "^1 



1/2 



S (p, ) = I rn.(N.-l )aT + 
' mn,(N,-l) L ' ' M 

I-"i^^(Er),] 



m(N -n,)ac + (N, 



where variance component estimators are 
dof i ned as fol lows : 



Let: 



^i.|' ^ik' Xijk' , and V(^j) 



i I 



be defined as in Equation (12). 



Then : 



.2 . N, . I 



rv - V 

' E, ^EI), 



■i N. 



m 



V - (h"i) V ■ 
^i TTT ^^'^i 



(EI), 



N. - I 

I 



(EI), 
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